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ABSTRACT 

For schools to spend Title I funds on comprehensive school 
reform (CSR) programs under the No Child Left Behind Act, there must be 
empirical evidence of significant improvement in the academic achievement of 
CSR students. Unfortunately, there is a dearth of convincing evidence that 
CSR programs have a positive impact on student achievement. Over a 2-year 
period, the effectiveness of five CSR models was assessed in 24 low- 
performing elementary schools in Kentucky, Tennessee, Virginia, and West 
Virginia. CSR models implemented at the schools included Success for All, 
Different Ways of Knowing, Balanced Early Literacy Initiative (a locally 
developed model), Direct Instruction, and Core Knowledge. Twelve CSR schools 
were matched with 12 control schools. Data were collected from teacher 
surveys, systematic classroom observations, and individually administered 
reading tests. Results indicate that CSR teachers used more direct 
instruction and performance assessment than did teachers at control schools, 
and urban CSR schools used significantly more ability groups, cooperative 
learning, and work centers than did rural CSR schools. Control schools used 
more independent seatwork than did CSR schools. Teachers at CSR schools were 
significantly more positive about school leadership, support, capacity, 
pedagogy, and outcomes than were those from control schools. Students at CSR 
schools outperformed those at control schools on two reading achievement 
tests. Rural schools had higher reading achievement scores than urban 
schools, consistent with the significantly higher use of sustained reading in 
rural schools. (TD) 
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President Bush’s recently enacted No Child Left Behind Act (HR 1 . PL 107-1 10 
(2002)) focuses on improving the achievement of low-achieving students in the nation’s 
highest poverty schools. It provides funds for eligible schools that employ proven 
strategies and methods for student learning, teaching, and management that are based on 
scientific research, so that all children can meet challenging state academic content and 
academic achievement standards. In pursuit of this goal, it funds comprehensive school 
reform (CSR) programs that encompass virtually all aspects of school operations 
including instruction, assessment, classroom management, professional development, 
parental involvement, school management, curriculum, and technology. In order for 
schools to spend Title I funds on CSR programs, the programs must have empirical 
evidence of significantly improving the academic achievement of CSR students 
compared to Control students. 

Although this is a laudable goal, in reality there is a dearth of convincing evidence 
that the implementatiqn of CSR programs has a positive impact on student achievement 
(Herman, 1999; Pechman & Fiester, 1996; Stringfield & Herman, 1994). In fact. 

Nunnery (1998) claims, “much of the literature on school reform is plagued with 
unsubstantiated claims based more on ideology and wishful thinking than on evidence 
(p.280). Herman (1999) summarized 24 nationally known school reform models and 
assessed the strength of the evidence related to their impact on student achievement. 

Only three models showed strong evidence of positive effects on student achievement. 
Another five models showed promising effects but the remaining models were marginal 
at best, and some models had no research at all. 

If educators are to be accountable for implementing school reforms that have 
evidence for improving academic achievement, then it is imperative for educational 
researchers to address for this lack of empirical evidence. The current study addressed the 
lack of empirical evidence with a systematic, longitudinal study of school-wide reforms 
in low-performing schools in the Southeastern United States. A matched control school 
multi-site replicated design (Slavin & Madden 1993) was used, consisting of 12 CSR 
schools and 12 Control schools. Schools were also categorized by geographic location 
(rural and urban) to determine if CSR implementation had a differential effect in Rural 
schools compared to Urban schools (Jennings, 1999; Leopold, Childers, & Howley- 
Rowe, 2000). 

Classroom observations at each school captured data related to instructional 
orientation and strategies, classroom organization, student activities, technology use, and 
assessment, which paralleled the categories specified in the No Child Left Behind 
legislation. Teacher surveys provided information on school climate, teacher 
expectations, and perceptions of outcomes. Student achievement was assessed via 
standardized reading tests that were individually administered to every student in the 
study. The primary rationale for including a reading assessment was that reading 
represents the most fundamental elementary school skill, which determines readiness and 
ability for mastering virtually all other school subjects. Secondarily, since each state in 
the study mandates a different test of academic achievement, comparing student 
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achievement across states is best accomplished by administering the same achievement 
test to all students in the study. 

Three research questions guided the study. 

1 . Is there a difference in classroom practices between CSR and Control 
schools after two years of implementation and do differences vary by 
school location (i.e. rural versus urban)? 

2. Is there a difference in school climate between CSR and Control schools 
after two years of implementation and do differences vary by school 
location? 

3. Is there a difference in reading achievement between CSR and Control 
schools after two years of implementation and do differences vary by 
school location? 



Method 



Twenty-four elementary schools in Kentucky, Tennessee, Virginia, and West 
Virginia participated in the research study. Each CSR school (12) was matched with a 
control school on the basis of free or reduced-price lunch, location (rural or urban), 
number of students and grade levels, and results on state-mandated tests. CSR models 
implemented at these schools included Success For All, Different Ways of Knowing, 
Balanced Early Literacy Initiative (a locally developed model), Direct Instruction, and 
Core Knowledge. 

Data were collected from multiple sources, and included measures of school 
climate (SCI© and SEPTQ©), systematic classroom observations (SOM©), and 
individually administered reading tests. The reading tests administered to students in the 
24 research study schools included: the Woodcock Johnson Reading Mastery Tests 
(Word Attack, Word Identification, and Passage Comprehension) (Woodcock, 1987) and 
the Durrell Oral Reading Test (Durrell & Catterson, 1980). 

Throughout the 1999-2000 school year, site researchers visited each school 
multiple times to conduct classroom observations. In the spring of 2000, site researchers 
administered surveys to the faculty, and trained testers administered the reading tests to 
first-grade students. During the 2001-2001 school year, site researchers again visited 
each school multiple times to conduct classroom observations. In the spring, teacher 
surveys and reading tests were again administered. By the end of the two-year period, 
approximately 3,500 classroom observations were conducted, teachers completed 1,100 
surveys, and 1,300 students completed reading tests. 



Results 

Classroom observations were analyzed via a three-way MANOVA using 
specified classroom practices as the dependent variable measures, and CSR status, 
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location, and year as the independent variables. Statistically significant results 0< 05) 
are reported using Effect Sizes ( ES) based on Cohen’s cl formula (Cohen, 1988). After 
two years of implementation, teachers at CSR schools were observed using more direct 
instruction (ES=+ 0.53) and performance assessment (ES=+ 0.35) than were those at 
control schools. Additionally, urban CSR schools used significantly more ability groups 
(ES= +0.33), cooperative learning (ES=+0.15), and workcenters (ES=+0.98) than did 
rural CSR schools. Control schools however, used significantly more independent 
seatwork than did CSR schools (ES=+0.35). 

The school climate data were analyzed via a MANOVA using the school climate 
dimensions as the dependent variable measures, and CSR status, location, and year as the 
independent variable measures. Teachers at CSR schools were significantly more 
positive about school leadership (£'iS' = +0.21) than were those from control schools. CSR 
teachers also reported significantly higher levels of agreement on issues related to support 
(ES=+ 0.55), capacity (£5'=+0.72), pedagogy (£5'=+0.63) and perceptions of outcomes 
than did those from control schools (ES=+0.57). Additionally, urban CSR schools tended 
to report higher levels of capacity than did rural CSR schools (£>S = +0.78). 

Reading achievement data were analyzed via a MANCOVA using 2001 reading 
achievement test scores as the dependent variable measures, CSR status and location as 
the independent variable measures, and student-level socioeconomic status (SES) and the 
previous year’s academic achievement (2000) as the covariates. Reading achievement at 
CSR schools was significantly higher than at control schools on the Woodcock-Johnson 
Word Attack test (£S=+0.14) and on the Woodcock-Johnson Passage Comprehension test 
(£S=+0.15) after student-level SES and previous year’s academic achievement were 
controlled. Analysis of reading test scores showed no differences between rural and 
urban CSR schools, but rural schools overall scored significantly higher than urban 
schools on the Durrell Analysis of Reading Difficulty (ZsS^+0.20). 



Conclusions 

Classroom practices seem to have been affected by the implementation of CSR 
models. CSR teachers used more direct instruction and performance assessment than did 
teachers at control schools, and urban CSR schools used significantly more ability 
groups, cooperative learning, and workcenters than did rural CSR schools. Because two 
CSR schools implemented Direct Instruction (which encourages direct instruction and 
ability groups) and five implemented Success For All (which encourages cooperative 
learning and ability groups), it is reasonable to expect differences in these classroom 
practices to emerge. Control schools on the other hand used more independent seatwork 
than did CSR schools, which is probably more reflective of traditional classroom 
practices. 

Teachers at CSR schools were significantly more positive about school 
leadership, support, capacity, pedagogy, and outcomes than were those from Control 
schools. These differences are understandable in light of the time and resources infused 
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into CSR schools, including extensive professional development, materials, and guidance 
on an ongoing basis. Control schools did not participate in these activities to the same 
degree as did CSR schools; hence, their perceptions were significantly lower on these 
dimensions. 

Student achievement also seems to have been influenced by implementing a CSR 
model. Students at CSR schools outperformed those at control schools on two reading 
achievement tests after SES and the previous year’s achievement were controlled. 
Because many of the CSR models in the study emphasized reading instruction, it is 
reasonable to assume that differences in reading tests are attributable in part to the CSR 
models. Additionally, higher reading scores at rural versus urban schools is consistent 
with the significantly higher use of sustained reading observed in rural schools. 



Implications 

Overall, CSR models together seemed to influence classroom practices, school 
climate, and achievement at CSR schools. If schools in this study are representative of 
all schools that implement CSR models, then it would be reasonable to expect similar 
increases in student achievement at other CSR schools. Before definitive statements can 
be made to this effect however, two questions must be addressed: (1) Will these results 
generalize to other schools that implement CSR models, and (2) Are the results obtained 
in the current study sustainable over time? Will schools in the current study revert back 
to their pre-CSR practices over time, especially after CSR funding is no longer available? 
A sound way to address these questions is to continue longitudinal studies of schools that 
implement CSR programs (Berends, et al. (2002a & 2002b), Fullan, 2000; Ross, et al. 
2002 ). 



At the policy level, policymakers and district-level educators who are considering 
implementing CSR programs in rural locations should address potential issues at rural 
schools regarding leadership, support, pedagogy, and capacity. Additionally, methods for 
capitalizing on the strengths inherent in rural schools (e.g., significantly higher levels of 
parent and community involvement in the educational process and perceptions of orderly 
student behavior) should be explored, especially because reading achievement was 
significantly higher at rural schools despite their limitations in other areas. 
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